1. Field of the Invention
This invention relates to a non-intrusive speech quality assessment system.
Signals carried over telecommunications links can undergo considerable transformations, such as digitization, encryption and modulation. They can also be distorted due to the effects of lossy compression and transmission errors.
Objective processes for the purpose of measuring the quality of a signal are currently under development and are of application in equipment development, equipment testing, and evaluation of system performance.
Some automated systems require a known (reference) signal to be played through a distorting system (the communications network or other system under test) to derive a degraded signal, which is compared with an undistorted version of the reference signal. Such systems are known as “intrusive” quality assessment systems, because whilst the test is carried out the channel under test cannot, in general, carry live traffic.
Conversely, non-intrusive quality assessment systems are systems which can be used whilst live traffic is carried by the channel, without the need for test calls.
Non-intrusive testing is required because for some testing it is not possible to make test calls. This could be because the call termination points are geographically diverse or unknown. It could also be that the cost of capacity is particularly high on the route under test. A non-intrusive monitoring application can run all the time on the live calls to give a meaningful measurement of performance.
A known non-intrusive quality assessment system uses a database of distorted samples which has been assessed by panels of human listeners to provide a Mean Opinion Score (MOS).
MOSs are generated by subjective tests which aim to find the average user's perception of a system's speech quality by asking a panel of listeners a directed question and providing a limited response choice. For example, to determine listening quality users are asked to rate “the quality of the speech” on a five-point scale from Bad to Excellent. The MOS, is calculated for a particular condition by averaging the ratings of all listeners.
In order to train the quality assessment system each sample is parameterized and a combination of the parameters is determined which provides the best prediction of the MOSs indicted by the human listeners. International Patent Application number WO 01/35393 describes one method for parameterizing speech samples for use in a non-intrusive quality assessment system.
There are particular issues to be considered when assessing speech quality over a packet switched network, in particular over Voice Over Internet Protocol (VOIP) networks.
In a VOIP network a VOIP gateway or edge-device converts packet data at an interface between a circuit switched network the VOIP network. The quality of the speech reconstructed by the edge-device from the packets it receives can be adversely affected by transmission degradation conditions such as variations in packet arrival time (jitter) and packet loss in the packet VOIP network. In general, the speech quality will reduce as jitter and/or packet loss increase.
The VOIP packet stream itself is well defined so VOIP calls can be identified either by monitoring call control signalling and extracting call set-up messages or by being able to recognise VOIP packets. A VOIP or packet based perceptual quality evaluation processor can be arranged to recognise VOIP packets as this enables calls to be identified even if the start of the call is missed. This technique also avoids problems when the packet stream and signalling information travel via different routes.
In order to monitor the speech quality of a VOIP from within the IP network, there is a need to account for the highly non-linear VOIP gateway.
A VOIP or packet based perceptual quality evaluation processor needs to account for each gateway according to the properties of the gateway because different gateway implementations respond to the effects of IP transmission in varying ways. Some areas of VOIP edge-device operation may be considered to be more interesting than others. These regions of interest are where small changes in transmission degradation conditions results in large changes in the perceived quality. Conversely if at a particular percentage packet loss the device consistently achieves low quality then the system quality is unlikely to improve with further packet loss and so testing for higher rates of packet loss is unnecessary.
As the performance profile of different edge-devices can vary significantly, it is highly advantageous to calibrate or configure the operation of a packet based perceptual quality evaluation processor so that it correctly takes the characteristics of the edge-device in the transmission link being assessed into account. However, calibrating a perceptual quality evaluation processor using an exhaustive set of pre-defined set of network conditions would either take too long or not provide detailed enough information in regions of interest. Similarly when testing the performance of a previously calibrated perceptual quality evaluation processor it is desirable to use a greater number of test sequences within regions of interest than outside regions of interest.
Although the primary application of the invention is the calibration and testing of a non-intrusive packet based perceptual quality evaluation processor, the very high levels of accuracy achievable by intrusive quality assessment systems make them ideal for performing the calibration and testing process.
A number of patents and applications relate to intrusive quality assessment, most particularly European Patent 0647375, granted on 14 Oct. 1998. In this invention two initially identical copies of a test signal are used. The first copy is transmitted over the communications system under test. The resulting signal, which may have been degraded, is compared with the reference copy to identify audible errors in the degraded signal. These audible errors are assessed to determine their perceptual significance—that is, errors that are considered significant by human listeners are given greater weight than those that are not considered so significant. In particular inaudible errors are perceptually irrelevant and need not be assessed.
The problem addressed by this invention is to produce a set of test sequences that exercise each edge-device over the main region of interest while keeping the testing time to a minimum.
2. Related Art
International patent Application No W00197414 describes an apparatus in which the perceptibility of degradations caused to signals transmitted over a transmission medium is measured by generating one or more predetermined transmission degradation conditions and subjecting a test signal to the transmission degradation conditions in a network simulation device. The degree to which each transmission degradation condition is perceptible to the human perceptual system is measured and a data set is generated and stored for converting one or more transmission degradation conditions to respective values of perceptibility. The data set may be a look-up table or an empirically determined formula. The data set may then be used on live traffic, by identifying objectively measured transmission degradation conditions in the received signal and retrieving from the data set in the data storage means a value of perceptibility associated with the transmission degradation conditions so identified. In this way a measure of the subjective significance of degradation in the signal can be derived from objective measures of degradation. An initial plurality of degradation conditions may be imposed on the system, and further degradation conditions are then selected according to the degree of variation in the resulting perceptibility measurement values.